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1.  Introduction 

Let  {Xq:  n  £  0}  be  a  Markov  chain  taking  values  In  a  countable  t 

state  space  E  ■  {  0,  1,  2,  ...  }.  If  {Xq}  Is  Irreducible  and 
positive  recurrent,  then  {Xq}  possesses  an  invariant  probability 
it  such  that 

l 

(1.1)  \  l  f(X.)  -  r(f)  -  /  f(y)  it(dy) 
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Pi  a.s.,  for  all  ic-integrable  f,  and  each  i  e  E.  Given  that 
(Xq)  represents  the  output  of  the  simulation  of  a  stochastic 
system,  the  simulator  is  often  interested  in  obtaining  point  estimates 
and  confidence  intervals  for  the  parameter  r(f).  The  problem  of 
constructing  such  estimates  and  Intervals  is  known,  in  the  simulation 
literature,  as  the  steady-state  simulation  problem  for  Markov  chains. 

One  approach  to  dealing  with  this  problem  is  to  exploit  the  fact 
that  {Xq:  n  ^  0}  is  a  regenerative  process;  the  process 
"regenerates'*  itself  at  return  times  to  a  fixed  state  1.  When  the 
sample  path  is  "blocked"  according  to  consecutive  regeneration  times, 
the  resulting  regenerative  intervals  are  independent  and  identically 
distributed  (i.l.d.).  CRANE  and  IGLEHART  (1974)  and  FISHMAN  (1973) 


proposed  techniques  for  the  steady-state  slaulatlon  problea  that 
exploit  the  1.1. d.  structure  of  the  above  regenerative  Intervals  to 
produce  a  methodology  that  bears  close  resemblance  to  the  classical 
statistical  procedure  for  1.1. d.  sequences.  This  regenerative  method 
for  analyzing  the  steady-state  of  countable  state  Markov  chains  has 
attracted  a  considerable  amount  of  Interest  since  Its  introduction; 
see  FISHMAN  (1978),  RUBINSTEIN  (1981),  and  LAN  and  KELTON  (1982),  for 
example. 

It  has  generally  been  assumed,  however,  that  the  regenerative 
method  has  only  limited  applicability,  due  to  the  belief  that  many 
stochastic  systems  chat  arise  in  simulation  are  not  regenerative.  In 
this  chapter,  we  shall  show  that  in  fact,  virtually  any  well-behaved 
almuatlon  has  a  regenerative-type  structure. 

As  has  been  mentioned  by  WHITT  (1980),  the  class  of  generalized 
semi-Markov  processes  (GSMP's)  appears  to  be  an  attractive  mathe¬ 
matical  model  for  the  general  discrete-event  slaulatlon.  These  pro¬ 
cesses  can  be  regarded  as  Markov  chains  taking  values  In  a  complete, 
separable  metric  space  (see  [24]  for  details  on  how  the  state  space  of 
GSMP  can  be  topologlzed  to  be  such  a  space).  After  dealing  with  some 
Markov  chain  preliminaries  in  Section  2,  we  shall  introduce  the 
concept.  In  Section  3,  of  a  "well-posed**  steady-state  simulation 
problem.  We  shall  prove  that  If  the  simulation  problem  is  well-posed 
for  a  Markov  chain,  then  the  Markov  chain  is  Harris  recurrent  with  an 
Invariant  probability;  the  converse  also  holds.  In  Section  4,  we 
shall  show  that  any  Harris  chain  can  be  embedded  in  a  weakly  regenera- 
tive  environment.  A  weakly  regenerative  process  Is  a  generalization 
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of  the  notion  of  regenerative  process  that  allows  the  process  blocks 
to  be  m-dependent,  as  opposed  to  independent  as  in  the  regenerative 


ease. 

In  [11],  a  steady-state  simulation  methodology  for  weakly  regen¬ 
erative  processes  is  developed  that  retains  all  the  attractive 
features  of  the  classical  regenerative  method. 

In  Section  5,  we  prove  that  the  assumption  of  a  wea^  regenerative 
embedding  for  a  Markov  chain  is  in  fact  equivalent  to  assuming  that 
the  steady-state  simulation  problem  is  well-posed  for  the  chain.  This 
result  shows  that  a  simulation  is  well-behaved,  in  a  certain  sense,  if 
and  only  if  the  process  is  weakly  regenerative.  Regenerative  struc¬ 
ture  is  therefore  the  norm,  rather  than  the  exception,  in  simulation 
problems.  This  has  Important  consequences  for  a  wide  variety  of  simu¬ 
lation  related  questions:  Quantile  estimation  (IGLEHAKT  (1976)), 
extreme  values  (IGLEHAKT  (1977)),  and  sequential  stopping  rules  for 
simulations  (LAVENBERG  and  SAUER  (1977))  are  among  the  examples. 

In  Section  6,  we  study  two  examples  which  show  how  a  Markov  chain 
can  fall  to  have  weak  regenerative  structure.  Section  7  considers  a 
necessary  and  sufficient  condition  for  the  simulation  problem  to  be 
well-posed,  and  shows  that  the  concept  of  well-posedness  is  equivalent 
to  assuming  a  certain  smoothness  on  the  transition  kernel. 

2.  General  State  Space  Markov  Chains 

Let  E  be  a  complete,  separable  metric  space,  and  £  its 
associated  Borel  sets.  A  function  P:  E  *  E  +  [0,1]  is  called  a 
probability  transition  kernel  if: 


1)  P(x,«)  is  a  probability  aaaaura  on  (E,E),  for  each  *  t  £, 

11)  P( • ,8)  la  an  E-aeasurable  function  for  every  B  e  E» 

The  n-etep  transition  probabilities  are  defined  through  the  relations 

P^x.B)  -  P(x,B) 

P^Cx.B)  -  /  PQ(y,B)  P(x,dy)  . 

E 

Let  Q  be  the  product  space  E  *  E  x  •••  and  £  the  associated 
product  onfield.  For  o>  -  (w^,  ...)  c  Q,  put  X ^(m)  ■  u^.  Then, 

for  each  probability  p  on  (E,E),  there  exists  a  unique  probability 
aeasure  P^  on  (Q,I?)  (see  IONESCU-TULCEA  (1949))  such  that-  for 

any  n  >  0  and  Bq  ,  . . . ,  Bq  €  E , 

(2.1)  e  Bq . XQ  *  Bq> 

-  /  p(dx0)  /  P(x0,dxl)  ...  /  P(xn_1,  dxn)  . 

B0  B1  Bn 

We  8 hall  commonly  write  P  to  denote  the  measure  P  ,  where  n  •  6 

X  X 

(6X(B)  *  1  or  0  depending  on  whether  or  not  x  e  B).  If  E^(«) 
corresponds  to  expectation  under  ,  then  it  can  be  shown  (see 
REVUZ  (1957),  p.  16)  that  the  measurability  of  the  kernel  P  allows 
one  to  write 

E^Z  -  /  ExZ  p(dx) 
for  any  bounded  ^-measurable  Z. 


4 


The  process  {Xq}  ,  constructed  relative  to  (Q,G,P^) , 
possesses  the  Markov  property.  To  be  precise,  let  0n:  Q  ♦  Q  be 
the  shift  given  by 

0Q(w)  -  (wn,  ...)  . 

Then,  for  any  bounded  F-measurable  Z, 

(2.2)  V*,9nl4»)  '  V  *»  *'*• 

where  -  B(Xq,  ...»  Xq),  the  c-field  generates  by  Xq,  ...,  XQ. 
Equation  (2.2)  is  a  statement  of  the  Markov  property  and  justifies 

< 

calling  {Xq}  the  general  state  space  Markov  chain  associated  with 
kernel  P. 


3.  Well-Posed  Steady  State  Simulation  Problems 

In  problems  where  the  decision  time  horizon  is  either  infinite  or 
very  large,  a  simulator  is  often  concerned  with  the  long-run  behavior 
of  the  sample  time  averages 


•  n+r  I  «V 

k-0 
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where  the  function  f  corresponds  to  some  performance  criterion  for 
the  stochastic  system  {Xq}  under  study.  The  simulator  expects 
that  ra(f)  converges  in  some  sense  to  a  number  r(f),  and  desires 
a  confidence  interval  for  r(f). 

Me  specialize  now  to  the  Markov  chain  context  and  suppose  that 
f  «  bjB,  the  class  of  real-valued  bounded  E-measurable  f.  Observe 
that  for  each  x  «  E,  the  sequence  {Ex  rn(f):  n  ^  0}  Is  then 
bounded.  Hence,  the  Bolzano-Ueierstrass  theorem  implies  that  all  sub¬ 
sequences  chosen  from  {Ex  ra(f):  a  0}  contain  a  further  subse¬ 
quence  which  is  convergent.  The  simulator's  expectation  of  the  system 
"settling  down"  then  requires  that  all  limit  points  must  coincide. 
Hence,  in  any  "reasonable"  simulation  of  a  Markov  chain,  we  must  have 

(3.1)  Ex  rQ(f>  -  r(f,x) 

r(f,x)  being  the  common  limit  point  constructed  above. 

Furthermore,  in  a  well-behaved  simulation,  the  limit  in  (3.1) 
should  be  Independent  of  the  initial  position  x,  as  one  would  hope 
that  the  simulator's  initial  condition  plays  no  role  in  the  conver¬ 
gence  issue.  This  motivates  the  following  definition. 

(3.2)  DEFINITION.  The  steady-state  simulation  problem  is  said  to  be 
well-posed  for  the  chain  {Xq)  if  for  all  f  c  bE,  there  exists  a 
number  r(f)  such  that 

(3.3)  E  r  (f)  -  r(f)  , 

x  n 
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the  convergence  holding  for  ell  x. 

One  point  requires  further  elaboration.  We  require  that  (3.3) 
occur  for  all  f  e  bE,  because  otherwise  the  simulator  would  have  to 
check  that  his  performance  criterion  f  lies  in  the  class  for  which 
convergence  holds,  and  this  is,  in  general,  unrealistic  to  expect.  We 
return  to  this  question  lster. 

Our  first  theorem  shows  that  chains  that  are  well-behaved  from  a 
simulation  standpoint  are  endowed  with  some  important  properties. 

(3.4)  THEOREM.  Suppose  that  the  steady-state  simulation  problem  is 
well-posed  for  {Zq}.  Then,  there  exists  a  P-invarlant  probability 
measure  x  on  (E,E)  such  that  for  all  x  e  E,  we  have 

rn(f)  -*■  r(f)  »  «f  S  /  f(y)  x(dy) 

Px  *•«.,  provided  x|f|  <  *. 

Proof.  Letting  f  range  over  the  class  of  indicator  functions  in 
(3.3),  we  get 

(3.5)  ST  jQ  W  ‘  «  *  r(V 
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for  all  Ac  E.  Sine*  the  set  function  on  the  left-hand  side  of  (3.5) 
is  a  probability  measure  for  all  n,  the  Vitali-Hahn-Saks  theorem 
(RENYI  (1970))  shows  that  Che  set  function  r(I^)  Is  also  a 
probability  measure,  say  x(»).  A  simple  argument  then  proves  that 
r(f)  “  iif,  allowing  us  to  re-write  (3.3)  as 

(3.6)  I  ( Pkf)(x)  ♦  *f  , 

where  (P,tf)(x)  -  /  f(y)  Pk(x,dy).  It  follows,  from  substituting 
the  bounded  function  g(y)  -  (Pf)(y)  in  (3.6),  that  it  must  satisfy 

xf  -  xPf 

for  all  f  c  bE,  and  hence  x  must  be  P-lnvariant. 

To  obtain  almost  sure  convergence,  we  first  observe  that  the 
shift  8  is  a  measure-preserving  transformation  under  Pn,  and  thus 
Blrkhoff's  ergodlc  theorem  (see  LAMPERTI  (1977),  p.  87-95  for 
definitions  and  results)  applies,  yielding 

(3.7)  r  (f)  -►  Z(f)  P,  a.s. 

•II  TV 

provided  x|f|  <  «  (Z(f)  is  the  bounded  invariant  r.v.  Ew{fg(Xg) ^1} , 
where  1^  is  the  class  of  shift  invariant  events).  Furthermore, 

Z(f )  “  xf,  Pn  a.s.  in  the  case  that  I  is  a  P^-trival  trfield 
(l.e. ,  all  events  in  1^  have  either  Pn  probability  0  or  1). 


8 


The  triviality  of  _I  follows  by  studying  the  P-haraonic 


functions.  A  function  g  is  said  to  be  P-harmonlc  if 

g(x)  -  (Pg)(x) 

for  all  x  c  1J.  Thus,  if  g  is  a  bounded  P-harmonlc  function,  (3.6) 
proves  that  g(x)  "  xg,  for  all  x  e  E,  and  thus  all  bounded 
P-harmonlc  functions  are  constants.  A  basic  result  in  Markov  chain 
theory  (see  e.g. ,  OREY  (1971),  p.  18)  yields  the  triviality  of  ^  as 
a  consequence. 

We  complete  the  proof  by  setting 

A  ■  (u:  r  (f,w)  +  xf) 

Q 

and  observe  that  Y  *  1^  is  a  shift  Invariant  event.  It  follows 
easily  that  h(x)  -  E*Y  is  a  bounded  P-harmonic  function,  and  thus 
h(x)  ”  P  (A).  But  (3.7)  proves  that  P  (A)  -  1,  yielding  P  (A)  -  1 

X  XX 

for  all  x  e  E.  I 

This  theorem  shows  that  convergence  of  expectations  turn  into 
almost  sure  convergence  provided  that  the  chain  is  well-behaved  from  a 
simulation  standpoint.  This  demonstrates  that  well-posedness  is  a 
rather  strong  requirement  to  impose. 

In  fact,  it  turns  out  that  the  concept  of  well-posedness  is 
equivalent  to  one  of  the  best  known  recurrence  concepts  for  general 
state  space  Markov  chains.  A  chain  (Xn)  is  said  to  be  Harris 
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v-recurrent  (sea  HARRIS  (1956))  if  there  exists  e  c-finite  measure  v 
on  (E,E)  such  that  v(A)  >  0  implies 


W 


1 


for  all  x  «  E.  Theorem  3.4  shows  that  a  chain  for  which  the 
simulation  problem  is  well-posed  is  Harris  it-recurrent .  A  converse 
is  also  available. 


(3.8)  PROPOSITION.  The  steady-state  simulation  problem  is  well-posed 

for  (X  }  if  and  only  if  {X  }  possesses  an  invariant  probability  x 
n  n 

and  is  Harris  x-recurrent. 

Proof.  Only  the  converse  needs  explanation.  By  Corollary  4.21,  the 
process  {Xq}  obeys  the  strong  law 

(3.9)  l  f  (X.)/ (n+1)  +  r(f ) 

k-0 

Px  a.s.,  for  all  f  c  bE  and  all  x  c  E.  The  bounded  convergence 
theorem,  applied  to  (3.9),  then  proves  Che  result.  I 


4.  Weakly  Regenerative  Structure  of  Well-Posed  Problems 

We  will  show,  in  this  section,  that  Harris  chains  can  be  embedded 
in  a  weakly  regenerative  environment.  This  will  be  accomplished  by 
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using  a  "splitting"  technique  due  to  ATHREYA  and  NEY  (1978)  and 
NUMMELIN  (1978).  Our  first  task  is  to  obtain  existence  of  a 
splitting"  measure  $. 

(4.1)  PROPOSITION.  Let  denote  the  set  of  pairs  (4  ,X.) ,  where 

i)  \(x)  is  an  E-measurable  function. 

ii)  P*(x,*)  ^  A(x)  4(»)  for  all  x  e  E, 

iii)  /  X(x)  v(dx)  >  0. 

Then.  is  non-empty  for  some  «,  provided  P  is  a  Harris 

kernel,  and  the  corresponding  Markov  chain  (Zq)  is  Harris 
v~ recurrent. 

Proof.  The  process  (Xn)  is  Harris  v-recurrent,  taking  values  in 
separable  metric  space  E.  Since  the  <r*field  E  is  therefore 
countably  generated,  Theorem  2.1  of  [20]  guarantees  existence  of  a 
C-set  C,  and  an  integer  m,  such  that  v(C)  >  0  and 

(4.2)  inf  p“(x,y)  ■  a  >  0  , 

(x,y)  c  CxC 

where  pm(x,*)  is  the  derivative  of  the  v-absolutely  continuous 
part  of  Pm(x,»)  with  respect  to  v.  Then,  (4.1)(j)  to  (iii)  are 
satisfied  by  $(•)  ■  v(C  n  »)/v(C),  and  \(x)  ■  alc(x)  v(C).  I 


The  following  result  will  also  be  necessary. 


(4.3)  PROPOSITION.  Ut  (*,  X(x))  t  Then,  there  exists  s  set 

ic  I  with  v(A)  >  0  such  thst: 

1)  lnf{X(x):  x  €  A)  -  X  >  0, 

11)  .  ?z{Xka  <  A  Infinitely  often)  "  1  for  ell  x  e  A. 

If,  In  addition,  (Zg)  possesses  an  invariant  probability  x,  then 

A  contains  e  subset  Aj  «  E  such  thst  v(Af)  >  0,  and 

iii)  sop  E  T  (n)  <  -,  where  T  (n)  “  inf{k:  X.  e  A.) 
x  e  Aj  x  *1  A1  1 

Proof.  Let  Ag  ■  {x:  X(x)  >  1/n},  and  observe  that  v(Ag)  >  0 
for  n  sufficiently  large,  since  /  X(x)  v(dx)  >  0.  We  now  recall 
that  all  Harris  chains  possess  a  cyclic  decomposition  C^,C2»..« .C^,  F 
such  that  (see  Theorem  1.3  of  [22],  p.  162): 

i)  C},  ...,  C4,  F  are  E-measurable  subsets  partitioning  E, 
with  v(F)  "  0, 

ii)  P(»,Ci+j)  “1  on  Ci  for  i  ■  1,  ...,  d-1  and  P(»,Cj)  •  1 
on  Dd, 

ill)  d  is  the  period  of  the  chain. 

Let  A  be  such  that  v(A  )  >  0,  and  observe  that  v(A  n  c.)  >  0 
n  n  n  l 

for  some  i.  Setting  A  *  Ag  n  Ci,  it  is  easy  to  verify  that 
{Xnd}  an  aperiodic  r-recurrent  Harris  chain  on  C^,  where 

t(»)  ■  v(C^  n  *).  The  chain  {Xnd>  on  therefore  has  a 

unique  o-finite  invariant  measure  and  t  <  ly  «  denotes 

absolutely  continuous  with  respect  to);  see  Theorem  2.7  of  [22].  One 
can  now  apply  Lemma  2.1  of  [19],  which  shows  that  the  skeleton 
{ Xgdffl:  n  0)  on  is  again  Harris  recurrent ,  with  invariant 
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d  dm 

measure  x^.  But  since  x^  Is  Invariant  for  P,  and  hence  P  , 

It  must  be  that  x^  “  x^  by  unldty  of  Invariant  measures  for  Harris 
chains.  Now,  %  <  x^a  so  x^a(A)  >  0,  and  therefore 

P  (X  .  c  A  infinitely  often}  ■  1 
x  Ham 

for  all  x  c  Cit  proving  (4.3) (i)  and  (11). 

Now,  suppose  {X^  possesses  an  invariant  probability  x. 

Then,  xd  ■  x(»  n  C^)/x(C^)  is  easily  shown  to  be  an  invariant 
probability  of  {X^}  on  ,  and  consequently  an  invariant 

probability  for  {Xndm}*  For  a  Harris  recurrent  chain  with  an 
invariant  probability  x,  the  set  of  1-regular  states  has  measure 
1  (see  [19],  Proposition  3.5(1),  for  definitions  and  details),  and 
hence,  by  Theorem  3.1  of  COGBURN  (1975),  the  chain  possesses  a 
strongly  uniform  subset  D.  Applying  this  result  to  ^  ^ 

obtain  existence  of  a  set  D  such  that  Xd(D)  >  0  and 

sup  Ex  Tjj(md)  <  •  . 
x  e  D 

Now,  the  set  of  1-regular  states  is  given  by  (Proposition  3.4,  1 5 J ) , 

R  -  {x:  Ex  Tp  <  -} 
and  thus  x,j(Dn)  >  x<j(A)  where 
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DQ  “  {*  «  A:  E%  Tp  <  n} 


r 

\  :  1 

By  Leans  3.2  of  (Si,  it  follows  that  for  a  sufficiently  large,  Dq 
is  strongly  unifora  for  {X^}  with  itd<Dn)  >  0.  This  Dq 
plays  the  role  of  A}  in  (4.3)(iii).  I 

Let  (4,X)  e  and  observe  that 

(4.4)  P^*(x,«)  -  X(x)  $(•)  +  (l-X(x))  Q(x,  •) 

where  Q(x,»)  ia  defined  via  (4.4).  Of  course,  0  <  \(x)  <  Pn(x,E) 

■  1,  and  hence  (4.4)  proves  that  the  signed  measures  Q(x, •)  are,  in 
fact,  probability  measures. 

In  the  case  m-  1,  Athreya,  Ney  and  Nummelin  have  applied  the 
decomposition  (4.4)  to  show  that  the  chain  {Xq}  may  be  regarded  as 
a  regenerative  stochastic  process.  This  allowed  them  to  simplify  the 
proofs  of  a  variety  of  classical  results  in  the  theory  of  Harris 
chains. 

However,  the  parameter  m  is,  in  general,  greater  than  one.  In 
that  event,  it  may  be  that  only  a  weakly  regenerative  embedding  of 
(Xq)  Is  possible.  To  describe  the  notion  of  a  weakly  regenerative 
process  (Xn),  defined  relative  to  a  probability  space  (Q,£,P),  we 
need  some  definitions. 

(4.3)  IAIiIITIOI.  A  random  time  T  is  a  O-measurable  function  from 


fi  into  the  non-negative  integers 


For  any  two  randoa  tines  T,,  T.  with  T,  <  T„,  we  define  fJ2 

12  1—2  — Ti 

to  be  the  o-field  generated  by  all  events  of  the  form  A  n  (Tj-T^-k), 
where  A  c  pjj4*  s  X^). 

(4.6)  DEFINITION.  The  process  (Zq)  la  said  to  be  weakly  regener¬ 
ative  of  order  a  If  there  exist  randoa  tlaes  0  *  Ijj  <  Ij  <  Tj  <  ••• 
such  that: 

T  -1 

1)  the  a- field  H  -  F_  are  n-dependent  for  soae  a. 

“fl  “X 

n 

l.e. ,  the  <r*f  lelds  Bq  and  ^  are  Independent  for 

i  >  «H, 

11)  for  1,  n  >  1  and  Jq  <  •••  <  j|, 

P{(X T  *  •***  W  *  Bj  Tn  "  J0 . Vi  “  V 

n  n 

-  ....  Z^)  «  B;  Tj  -  J0 . Vi  "  Jl} 

for  each  B  In  the  product  o-field  E^,  where  x  -  T. ,-T, . 

—  n  x 

AAA 

We  now  turn  to  construction  of  a  probability  space  (Q.JF.P^), 
which  supports  the  chain  {Xj,}  under  initial  distribution  p,  such 
that  the  Harris  process  (Xn)  is  weakly  regenerative  of  order  1. 
Recall  that  E  is  a  complete,  separable  metric  space,  and  thus,  so  is 
the  product  space  E®  under  the  product  topology.  Furthermore,  the 
separability  of  Em  implies  that  the  Borel  sets  under  the  product 
topology  coincide  with  the  product  a-field  _Em  (see  DELLACHERIE 
and  METER  (1978),  p.  9).  Thus,  there  exist  regular  conditional 
distributions  Px(y,dz)  (see  BREIMAN  (1968),  p.  79,  401)  such  that 
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px(b)  -  /  I /  iB(*,y)  Px(y»<J*)J  p“c*,dy) 


for  all  B  e  E®. 


Suppose  now  that  ^  is  non-empty,  and  that  (4,1)  « 

Let  E*  -  u"  x  A,  E  •  EB  *  {0,1},  Q  •  E*  *  E  x  E  x  ...  ,  and 

j*U  B  B  B 

* 

take  £  as  the  corresponding  Borel  product  <r*field  (treat  E*  as 
the  topological  sum  of  EJ  x  A,  as  J  ranges  from  0  to  «•). 


Denote  the  coordinate  projections  on  Q  by  Yq,  (1^6^),  • m 

A  A  A 

and  construct  a  probability  measure  on  (Q,,P)  as  follows. 

Writing  Yq  -  (Y0<0),  ...»  YQ(N)) ,  Y^  -  (Y^l),  ....  Y^m)),  **  set» 
for  B  c  Ek+1 


(4.7)  P^{Y0  c  B;  N  -  k>  -  P|i((X0,  ....  X^)  £  B;  -  k) 

where  SA  -  Inf  {k  0:  c  A)  ,  and  A  Is  the  E-measurable 

subset  of  (4.3) (1)  and  (11)  corresponding  to  (4,1).  Recall  that 

A 

v(A)  >0  so  that  P^{SA  <  «•}  -  1,  and  hence  (4.7)  defines  P^  as  a 
probability  on  E*.  Also,  for  x  e  E  and  B  £  E®,  let 

P(x,  B  x  {0})  -  (1  -  l(x))  /  [/  IgCv.u)  Px(u,dv)]  Q(x,du)  , 

(4.8) 

P(x,  B  x  {1»  -  X(x))  /  [/  IjU.u)  Px(u,dv)]  4(du)  . 

For  any  x  c  E,  we  can  then  define  a  probability  measure  Px  on 
Ej  *  E,  x  •••  by  the  formula 
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\C»1  6  ®i»  •••»  €  ®n»  •••»  “  *■„) 

-  /  p(x,  &y1  x  {ip)  ...  /  Hy^OO,  <iyn  *  {iQ»  ; 

Bi  Bn 

existence  of  such  a  measure  follows  by  the  theorem  of  lonescu-Tulcea. 

A 

The  measure  is  then  given  by 

(4.9)  yY0  c  B0,  N  -  k,  (Y^  6r  ...»  Yq,  6fl)  c  Bj) 

”  I  VY0  «  dV  N1  "  k>  5yn(k)«V  51 . Yn*  V  c  V* 

B0  0 

Finally,  we  define  the  process  {Xq)  ,  on  the  probability  space 

AAA 

(Q.^.P^),  by  the  formula 

XQ  -  YQ(k);  k<N 

XWkm+j  "  \+l^;  1  -  3  -  m  * 

(4.10)  PROPOSITION.  The  process  {Xq}  has  marginal  distribution 
P^  on  (Q,F)  -  (E-,  £"). 

Vc+1  n 

Proof.  For  Bj  e  _E  ,  Bj  «  E_  ,  it  is  sufficient  to  prove  that 

i  •••*  ®  Bl*  (*k+i ♦  •••»  ^it+n^  *  B2*  ® 

**  •  ••»  *  Bj,  ®  ®2*  ^  • 
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But  this  follows  from  (4.7)  and  (4.8),  upon  observing  that  for 


B  c  E“, 


P(x,B)  -  /  1/  ^(v.u)  P^u.dv)!  P(x,du) 

-  P,(tt. . X)  e  B}  .  I 

xi  n 


Let 


Tx  -  lnf{k  >1:^-1} 


Tq  -  inf{k  >  Ta_j:  ^  -  1} 

(4.11)  PROPOSITION.  The  process  {(T#iy:  a  >  1)  is  weakly 
regenerative  of  order  1  with  respect  to  the  randoa  tlaes  {T^ :  j  >  2} , 
aider  the  probability  ,  for  all  x  e  A. 


Proof. 

Now, 


We  first  need  to  shown  that  the  T  ' s 

II 


are  finite  P^  a.s. 


(4.12)  PX{T,  -  -)  <  K,(6  ,  -  0 . 5,  „  -  0) 

1  n 

where  R_  *  0,  R ...  -  inf{k  >  R  :  Y.(m)  c  A).  Now,  recall  that 
u  nrl  n  K 

e  A  infinitely  often,  provided  Xq  e  A.  Hence,  by  Proposition 
4.8,  the  R^'s  are  all  finite  P  a.s.,  and  thus,  by  the  strong  Markov 
property  applied  at  time  Rq,  we  can  bound  (4.12)  by 
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— -  ‘^Sjaamw-.-  '.in  ilnABBiiir 


(4.13) 


Ex{6R,+1  "  °»  •**.  6R  .+1  "  °»  PY_  (a){6l  " 
i  n-i  k 

n 


<  (l-K)  ix{«  -  0 . «„  ,  -  0) 

1  n-1 


since  (l-X(y))  <_  l-X  for  y  e  A.  Repeating  this  process  n-1  more 
times  shows  that 


yTi  -  •}  <  a-x>n 

for  all  n,  and  hence  <  <*>,  a.s.  A  similar  proof  shows  that 
T  <  ®,  P  a.s.  for  all  n. 

Q  X 

Now,  let  be  the  o-fleld  generated  by  events  of  the  form 

A  n  {02~Di  «  k},  where 


A  €  . ^Uj+k’  ^Oj+k^  * 

and  are  random  times.  Let  F  be  the  o-fleld  B  x  E  x  •  •• 

and  let  8  be  the  shift  on  E  xE  x  •••.  Then  (Y,t)»0  ■  ((Y_,t,), 

n  m  2  2 

...)  where  (Y,x)  ■  ((Y^.-tj),  (Y2,x2),  ...).  For  B  e  F, 
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T  -1 

(4.14)  Px{(T,x)*eT  «  BjGj0  } 

B+l 

-  v v<Y*>*Vl  «  B|<h“>&a’l> 

-  -  -  T  -i 

-  E  (P  {(Y,t).8_  «  B}|G.n  } 

X  YTn(»)  T1  ,_1 

-  /  P  {(Y,t)»0_  c  B}  *(dx) 

provided  that  one  can  show  Chat 

T  -1 

VYX  (m)  c  A|GiD  >  "  *(A) 

n 

But  this  follows  from  the  fact  that 

Px{(Yq,  ...»  Yk-1)  c  B,  Yk(m)  «  A,  Tn  -  k} 

-  Ex{$U)  XCY^jCm));  (YQ . Yw)  e  B;  k> 

-  *(A)  Px{(Yq . Yk_1)  e  B;  T  -  k>  . 

Equation  (4.14)  shows  that  j^_i  and  Bq+i  are  independent,  and 
have  the  "shift"  property  (4.6)(ii),  proving  the  result.  I 
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Because  of  Che  form  of  P^,  It  Is  now  a  simple  matter  to  verify 
that  Tq,(Yj,6j),  •••  is  itself  weakly  regenerative  of  order 

A 

1,  under  the  probability  P^. 

A  weakly  regenerative  process  {Xn}  is  said  to  be  positive 
recurrent  if  Et2  <  ■».  We  will  now  extend  Propositon  4.11  so  as  to 
address  the  recurrence  nature  of  the  weakly  regenerative  embedding  of 
a  Harris  chain. 

(4.15)  PROPOSITION.  Let  {Zq}  be  a  Harris  chain  with  Invariant 
probability  x.  Then.  (Xq)  can  be  embedded  in  a  probability  space 

a  a  a 

(Q  ,_P»P^>  on  which  {Xq}  is  positive  recurrent  as  a  weakly 
regenerative  process. 

Proof.  We  shall  prove  that  for  any  ($,\)  e  the  weak  regenera¬ 
tive  embedding  of  (4.7)  through  (4.8)  yields  a  positive  recurrent 
weakly  regenerative  process. 

Let  Aj  =_  A  be  as  in  (4.3)(iii),  and  set  Uq  -  0, 

0*  -  inf{ j  >  0,^:  Yj (m)  c  Aj}  . 

Observe  that  for  x  e  Aj, 

\\  <  k  sup  Ex  Ta  (m)  H  kg  <  »  . 
x  e  Aj  1 

Then,  in  particular. 
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V '?,(.)  °i>  < 'A 


so  chat 


/  EyUj  *(dy)  <  ExU2/X  <  - 
for  x  «  Ay.  Hence,  for  x  e  A,  we  have 


(4.16)  Va  *  V*iT  i  /  Vi  ♦<ta> 

xi 

<Ey02A*/Bi(lf)j(.)(T1)(  *<ta)  . 

where  y  c  Aj  Is  fixed.  How,  given  x  e  Ay ,  it  is  evident  that 

(4.17)  ixTl  <  E,{0k;  60i+,  -  0 . «Vi+i  ■  0.  6^  -  1}  +  1 

i  j,  f.(0k;  \«  -  0 . \.,+l  "  °)  +  1  • 

Applying  the  strong  Markov  property  at  time  U^-y  shows  that  the 
k'th  summand  in  the  .last  expression  is  just 
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Sx^°k-1  PYU  (b){61  “  \+l  "  °*  ***’  60k_2+1  " 


+  61  *  °>*  “  °*  -■  \  ,+l  *  °> 

Uk-1  1 


which  Is  clearly  bounded  by 


<»-»  'JVl'  \+l  -  0 . \_2+l  ■  °> 


♦  *  \l\*i  ■  0 


t  •  •  •  t 


\-2+1 


0] 


<  (l-\)  Ex{0k-1;  60^+1  -  0 . 6\_2+1  " 


+  p(l-X)k”2  ; 


for  the  second  inequality,  see  (4.13).  Repeating  this  process  k-1 
more  times  shows  that  the  kth  summand  of  (4.17)  Is  bounded  by 
P(l-X)*t"2f  and  hence 


E  T -  <  2p  +  l  (k+2)  p(l-X)k  +  1  . 

0 

Since  this  bound  is  uniform  over  x  e  A,  relation  (4.16)  proves  that 
E  i,  <  •  Is  uniformly  bounded  as  well,  on  x  c  A.  The  form  of 

X  £  ^ 

then  dictates  that  <  • ,  proving  the  positive  recurrence.  I 
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*1*  > 


Given  Proposition  4.15,  we  immediately  obtain  the  following 
corollary,  the  proof  of  which  is  based  on  the  strong  law  for  weakly 
regenerative  positive  recurrent  processes  (Theorem  3.1,  [10]). 

(4.18)  COROLLARY .  If  {Zq:  n  ^  0}  is  a  Harris  chain  with 
invariant  probability,  then 


l 

k-1 


f(V/ 


n  +  E 


Vf)/V2 


P  a.s. 
x 


for  all  f  «  bK,  where 


Y  (f) 
n 


f(V  . 


5.  Harris  Chains  and  the  Weak  Regenerative  Property 

In  Section  4,  we  showed  that  Harris  chains  with  invariant  proba¬ 
bility  can  be  embedded  in  a  positive  recurrent  weakly  regenerative 
environment.  In  this  section,  we  will  show  that  the  converse  holds. 

(5.1)  DEFINITION.  A  subset  Ac  I  is  said  to  be  stochastically 
closed  If  P(x,A)  ■  1  for  all  x  e  A. 

For  stochastically  closed  A,  the  restriction  to  A  is  the  chain 

A 

with  transition  kernel  P(x,E)  "  P(x,  E  ft  A),  where  x  ranges  over 
A.  The  following  theorem  is  a  precise  ststement  of  the  converse. 
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(5.2)  THEOREM.  Let  P  be  a  transition  kernel  and  p  a  probability 
on  (E,£).  Suppose  that  there  exlsta  a  probability  space  (Q,G,P) , 
supporting  a  sequence  {Xq},  such  that: 

i)  the  marginal  distribution  of  {Xq}  is  P^, 
ii)  the  process  {Xq}  is  positive  recurrent  and  weakly 
regenerative  of  order  m. 

Then.  P  possesses  a  stochastically  closed  subset  A  such  that  the 
restriction  to  A  is  a  Harris  chain  with  invariant  probability. 

The  main  tool  required  in  proving  this  result  is  the  following 
proposition. 

t 

(5.3)  PROPOSITION.  Under  the  hypotheses  of  Theorem  5.2,  P  possesses 
an  invariant  probability  %  and  a  set  B  e  IS,  with  x(B)  **  1,  such 
that 

(5.4)  E  r  (f)  •*  xf  ,  for  all  x  e  B  . 

x  n 

Leaving  the  proof  of  this  result  aside  for  the  moment,  we  can 
prove  Theorem  5.2. 

Proof  of  Theorem  5.2.  Let  B  be  as  in  Proposition  5.3  and  put 
Bq  -  B.  We  then  define  the  decreasing  sequence  of  sets  Bfc  by 
the  iteration 


Bk+1  "  e  Bk:  p(x»  BkJ  "  1}  ' 
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Observe  ChaC  since 


*(B)  ■  /  P(x,B)  x(dx)  , 

P(x,Bq)  “  1  for  x  a.e.  x,  and  hence  x(B})  ■  1.  A  simple 
induce ion  proceeds  Co  show  ChaC  xCB^)  -  1  for  all  k.  Let 
A  ■  n£  B^.  Then  x(A)  ■  1  and  P(x,A)  “  1  for  all  x  c  A,  proving 
chac  A  is  stochastically  closed.  Furthermore,  (5. A)  implies  that 
the  steady-state  simulation  problem  is  well-posed  for  the  restriction 
on  A,  and  thus,  the  theorem  follows  by  Proposition  3.8.  I 

For  Che  proof  of  ProposiCion  5.3,  we  will  reduce  the  analysis 
from  that  of  a  weakly  regenerative  process  to  that  of  a  family  of 
regenerative  processes,  l.e.,  weakly  regenerative  of  order  0. 

Assuming  that  {Xq}  is  weakly  regenerative  of  order  m,  let 
{W^:  n  ^  0}  be  processes,  defined  as  follows: 

lf  i<T*(k)+l 
if  i  <_  k,  j  •  1 ,  . .. ,  m 

0  * 1  <  \u>  • 

where 
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V°>  -  Wl  +  J0  \<0) 

r 

Y_(J)  ■  k  +  I  T  .  j  •  1.  •••>  “ 

r  8-0  V3) 

T1s(j)  ■  X(k)  +  1  +  a(afl)  +  j  . 

Basically,  Che  process  wjj  records  Che  process  values  of  X 
over  weak  regeneraclve  epochs  wlch  assoclaCed  Indices  of  Che  form 
s(irH)+j ,  modulo  a  common  pordon  of  lengch  k+1  ac  Che  beginning. 
InCuiCively  Chen,  Che  individual  processes  W3  are  regeneraclve. 

i 

(S.S)  LEMMA.  For  j  -0,  1,  ....  m,  the  processes  {W^:  n  ^  0}  are 
regenerative,  with  respect  to  the  random  times  {yr(j);  r  1}. 

Proof.  He  prove  only  the  case  J  -  0,  the  other  processes  being 
similar.  Let  A  e  _En,  Be  EP,  and  consider  for  n  k. 


P{(Wq,  ...,  c  A;  y^(0)  -  n; 

(“2 . 72<0>  -  »> 


p((Xq»  •••.  Xq_j)  c  A;  T 


(X, 


\<0) 


l(k)+2 


\co>«*-*> 1 V°> 


P} 


l  l  P(A,  r  r  "  B  ) 

-0  ri<  k  J,rl’  2  3 * rl  ,r2 


J"°  /<£< 
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where 


^,rrr2  "  ^X°’  ***’  X“_1*  £  A;  TJ  ri*  Tj+1  "  r2*  Tj+2 


n> 


j+l+(m+l) 


*  •  •  •  9 


Vl+C-HJ+J-l’  ‘  ‘‘VW-HJ  '  1,1 


Of  course,  A  e  v^+*  H  ,  the  <j-field  generated  by  H,.,...,!! 

J»rl*r2  <1“0  — q  —0  -j+l 

whereas  B  c  H,.,,,  ...»  and  hence  the  two  events  are  indepen- 

J,ri,r2  -j+l+(m+-l)’ 

dent  by  m-dependence.  It  easily  follows  that  the  generating  sets  of 
the  regenerative  o-fields  associated  with  {W^}  are  independent; 
property  (4.5)(ii)  follows  through  a  similar  argument..  I 


It  is  straightforward  to  show  that  the  regenerative  processes 
{W^}  are  positive  recurrent  if  the  process  {Xq}  is  positive  recur¬ 
rent  as  a  weakly  regenerative  process.  Consequently,  Proposition  5.9 
of  [10]  can  be  applied  to  show  that  if  f  c  bjJ,  then 


(5.6) 


«  n+k 


*{  L  f(wJ)/n|wJ,  ....  w£}  -  ETjCfsJJ/STjCj) 


<  »fi  ZB(j) 


where  If  I  *  sup{|f(x)|:  x  c  E} ,  Z„(J)  is  a  random  variable, 
independent  of  f,  such  that  Z^C j )  >0  a.s.,  and 


(5.7) 


-  I,..  «<>  -  l 


Ti(k)+i+j+r1 


n»T. 


n-Yl(J)  “  ‘Kk)+l+j 

TjO)  -  Yjdjj)  . 


f(*„)  * 
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The  representation  {5.7),  In  terms  of  XQ,  Implies  that  EYj(f;J)/ 

A 

Et^(J)  Is  Independent  both  of  j  and  k;  denote  the  common  value  by 
itf.  We  are  now  ready  to  prove  Proposition  5.3. 


Proof  of  Proposition  (5.3).  Let  Aj(n)  be  the  number  of  Xr, 
from  the  collection  {X^+^,  X^u) •  recorded  by  W^.  Then, 


n+k 

l  f(X  )/n 
r-lc+l  J 


m 

I 

j“0 


Jlj  (n)+k 

l 


r-k+1 


f(wj)/n 


so  that 


(5.8) 


«  n+k 

E{  I  f(X  )/n|xQ . X.  }  -  *f 

r-k+1  r  1  0  K 

|  *  *  Aj(Q)+k 

-  I  I  E{  l  f<wj)/n|wj . wj}  -  wf 

lj-0  r-k+1  r  '  u  K 


Now,  it  is  easy  to  prove,  using  m-dependence,  that 


rn(3)  -  | Zj Cn)/n  -  1/ (m+1) |  *  0 

A 

P  a.s.  We  can  therefore  find  constants  a  (j)  4-0  such  that 

Q 
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<  tu 


P  a.#.; 


see  Lemma  5.14  for  a  proof.  Then,  by  Theorem  9.4.8  of  CHUNG  (1974), 

» 

(5.9)  P{rn(j)  >  «nO)|wJ . w£}  -o 

a.s.  Thus,  we  have  that 

Xj(n)+k 

E{  l  f(W^)/n|w4 ,  ....  wj}  -  nf/(m+l}| 
r-k+1  r  1  u  *  I 

b  -He 

<  E{  l  f(wJ)/n|wj[ . wjl  -  itf/ (m+1 } I 

r-k+1  r  i  u  it  | 

+  an(j)  ifi  +  ifi  p{rn(j)  >  »nO)|wJ . «k> 

<  Ifi  (ZQ(j)  +  aQ(J)  +  P{rn(j)  >  an(J)|wJ . w£» 

by  (5.6)  and  (5.9),  where  ba  is  the  greatest  integer  less  than  or 
equal  to  n/(m+l).  It  therefore  follows  from  (5.8)  that  there  exists 
a  sequence  Kq  of  random  variables.  Independent  of  f,  such  that 

*  n+k 

(5.10)  |e{  l  f(Xr)/n|x0,  ...,  X^}  -  itf  I  <  Ifi  Rn  a.s. 

r-k+1 

A 

where  R  ♦  0  P  a.s.  Let 

Q 

t  (x,*)  -  l  Pk(x,»)/n 
r-i 
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<  PL. 


and  observe  that  (5.10)  Implies  that 


(5.11)  |WA)  "  «(A)  |  <  Rn 

a.s.  Nov,  _E  is  the  Borel  field  of  a  separable  metric  space,  and 
therefore  generated  by  a  countable  algebra,  call  it  A.  By  (5.11), 

A 

the  set  Ay  has  P  measure  1,  where 


Ay  *  {<*>:  l-CnCx^Cw) ,  A)  -  it(A)  |  <  RQ(u),  for  all  A  c  A,  n  >  1}  . 

A 

Hence,  P{Xy  e  D)  ■  1,  where  D  is  the  ^-measurable  set 

9 

D  -  {x:  sup  It  (x,A)  -  it(A)  I  -*•  0}  . 

A  <s  A  1  ' 

We  can  then  apply  Lemma  5.15  to  any  x  c  D,  to  obtain  the  result  that 

(5.12)  /  f(y)  xQ(x,dy)  ♦  itf 

over  all  f  e  bE  simultaneously,  provided  x  c  D.  Now,  relation 

(5.12)  can  be  re-written  as 


E  r  (f)  uf  , 
x  n 

and  hence,  the  proof  of  the  result  is  complete,  provided  that  we  show 
x(D)  •  1.  By  the  strong  law  for  positive  recurrent  weakly  regenera¬ 
tive  processes  (Theorem  3.1,  [10]),  and  bounded  convergence. 
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_ jit** 


?  * 

l  P{X  6  A)/n  -►  x(A) 
k-1  * 

and  therefore  It  la  clear  that  *(•)  Is  absolutely  continuous  with 

i 

respect  to 

(5.13)  l  P{X.  e  •>  2"<k+1)  . 

0 

A 

But  P{3^  «  D  for  all  k}  ■  1,  and  hence  D  has  full  measure  with 
respect  to  the  probability  given  by  (5.13).  I 

(5.14)  LEMMA.  Suppose  that  a  sequence  rn  ♦  0  a.s.  Then,  there 
exists  constants  ig  >  0  such  that 

p{r  >  a  infinitely  often)  *0  . 

n  o 

Proof .  Let  mg  -  0  and  define  by 

m^  "  lnf{m  >  m^jS  P{rQ  <  1/k  for  all  n  m)  ^  1  -  2  k)  . 
For  i%  <  j  <  afc+i ,  put  aj  *  1/k.  Then 

•  "if 

P{r  >  a  Infinitely  often)  <  £  P(e£)  <  J  2  <  •  , 

11 

where  E^  -  {r  <,  1/k  for  n  ^m^}.  Then,  the  result  follows  from 
the  Borel-Cantelli  lemma.  I 
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(S.1S)  UN (A.  Ut  tn,  t  be  probabilities  on  •  aunrtblt  space 
(1,1),  and  lac  Item  algebra  which  ganaracaa  E.  Than,  if 

(5.W)  tup  |t_  (A)  -  x(A)  I  *  0 

At A ’ “  1 

ona  alao  baa 


x^t  *  xf  for  all  f  e  bE  . 


Proof.  Lac  A  represent  the  operation  of  symmetric  set  difference. 
We  first  need  to  show  Chat  If  B  e  E,  then,  for  all  e  >  0,  there 
exists  A  e  A  such  that  vj(A  A  B)  <  e  (l.e. ,  B  can  be  approxi- 
nated  by  sets  c  A),  where  vj  (1  *  1,2)  are  probabilities  on 
CEE). 

We  proceed  via  a  “monotone  class'*  argument.  Let  M  -  {B  e  E:  B 
can  be  approximated  by  sets  e  A) ,  and  note  that  A  =  M,  and  that  M 
Is  closed  under  Increasing  unions  and  decreasing  Intersections.  The 
monotone  class  theorem  ([7],  p.  13)  then  shows  that  M  »  IE. 

With  this  result  In  hand,  observe  that  for  B  c  E,  there  exists 
Bq  c  A  such  that  rn(B  A  Bn)  <  e,  r(B  A  Bn)  <  c.  Then, 

K<»>  -*<»>| 

■  K»>  -  Wl  +  |',«,)  -  *«n>|  ♦  -  ’«„>l 

<  2e  +  sup  I t  (A)  -  t(A)|  ♦  2e 
As  A  1  1 
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and  hence  tn(A)  *  t(A)  for  all  Ail.  For  f  e  bE,  approxlaate 
f  uniformly  by  indicators  such  that  I f- <  1/k.  Then, 

since  we  already  have  established  convergence  for  simple  functions, 

Iv  -  "I  <  2,‘-V  *  IVk  -  «k|  *  2,f-v 

which  proves  our  result.  I 

Theorem  5.2  shows  that  if  a  chain  can  be  embedded  in  a  weakly 
regenerative  environment,  then  the  chain  ip  Harris  recurrent.  On  the 
other  hand,  in  Section  4,  it  was  shown  that  Harris  chains  can  be  made 
weakly  regenerative  with  respect  to  random  times  that  are 
nonanticipating  with  respect  to  the  chain.  Hence,  we  may  conclude 
that  if  a  weak  regenerative  embedding  is  at  all  possible,  then  it  can 
be  done  via  random  times  that  are  nonanticipating. 


More  on  the  Steady-State  Simulation  Problem 


In  the  last  three  sections,  we  have  shown  that  the  assumption  of 


well-posedne8S  is  essentially  equivalent  to  either  of  the  following: 

1)  assuming  that  the  chain  is  Harris  recurrent, 

11)  assuming  that  the  chain  has  a  weakly  regenerative  embedding. 
In  view  of  the  strength  of  the  well-posedness  assumption.  It  is 
Incumbent  upon  us  to  Investigate  the  concept  somewhat  further.  Before 
proceeding,  a  little  reflection  shows  that  a  minimal  condition  for 


discussion  of  Che  steady-state  simulation  problem  is  existence  of  an 
initial  probability  p  and  a  limiting  probability  v  such  that 

(6.1)  l  f  Pk(x,0  p(dx)/n  ->  v(*) 
k-1 

where  ■>  denotes  weak  convergence  of  probabilities  (recall  that  E 
is  a  metric  space  so  this  makes  sense).  Now,  virtually  all  chains 
that  arise  through  applied  probability  considerations  have  kernels  P 
that  are  Feller,  i.e.,  if  f  e  bC  (bounded  E  continuous  functions), 
then  Pf  e  bC.  The  following  result  is  well  known;  we  give  a  proof 
for  the  sake  of  completeness. 

(6.2)  LEMMA.  If  P  is  a  Feller  kernel  and  (6.1)  bolds,  then  v  is 
^-Invariant . 

Proof.  For  f  c  bC,  (6.1)  Implies  that 

n  . 

I  /  (P  r)(x)  p(dx)/n  -*•  vf 
1 

But  Pf  c  bC  as  well,  so  we  also  get 

n  . 

I  J  (PKf)(x)  n(dx)/n  vPf  . 

1 

Hence  vf  *  vPf  for  all  f  c  bC,  proving  that  v  is  P-invariant.  I 
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The  above  discussion  suggests  that  we  limit  our  stud;  of  the 
slmultion  problem  to  chains  that  have  a  unique  invariant  probability. 
In  the  remainder  of  this  section,  we  consider  two  examples  of  chains 
that  illustrate  how  the  well-posedness  assumption  can  be  violated, 
even  in  the  presence  of  such  a  probability. 

Consider  the  chain  {Xq:  n  ^  0}  on  state  space  E  -  {0,1,2,...} 
associated  with  transition  matrix  given  by 


P 


00 


1 


p«  •  5i+i.j 


for  1  >  1 


The  simulation  problem  is  not  well  posed  for  {Xn}  since  for 
f(y)  ■  6oy,  we  have 


h  -  0  *  ‘  -  *0  ra(f) 

when  i  >  1,  violating  (3.3).  The  difficulty  here  is  that  part  of  the 
state  space  is  transient.  The  situation  is  saved  by  restricting  the 
chain  to  the  subset  {0}  — the  simulation  problem  is  then  well-posed 
for  the  restricted  process. 

One  might  hope  that  this  can  be  done  in  general.  Specifically, 
the  existence  of  a  unique  invariant  probability  x  forces  the  shift 
0  to  be  a  measure-preserving  ergodlc  transformation  on  (Q.P.P*) 

(see  ASH  &  GARDNER,  p.  141)  for  a  proof  of  the  ergodicity).  Applica¬ 
tion  of  Blrkhoff's  ergodlc  theorem  proves  that  for  each  AcE  with 
x(A)  >  0,  there  exists  a  set  c  E  for  which  x(N^)  ■  0  and 
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(6.3) 


P  a.s 
x 


CO 

s 

lt»0 


W  m  - 


for  x  i  N^.  If  one  could  find  a  common  ir-null  sec  N  such  Chat  for 
x  i  N,  (6.3)  holds  for  all  A(  E  with  n(A)  >  0.  then  one  could  hope 
Co  argue  Chat  Che  chain  restricted  to  the  complement  of  N  is  Harris 
recurrent.  Such  a  result  would  therefore  prove  that  a  chain  with 
unique  invariant  probability  always  possesses  a  restriction  for  which 
the  simulation  problem  is  well-posed. 

However,  this  is  too  much  to  expect,  in  general.  Consider  the 
autoregressive  process  on  E  ■  [-1,1}  defined  by 


(6.4) 


Xn+1  "  2  Xn  +  errt-l 


where  {en:  n  >  1}  is  Independent  and  identically  distributed  with 
P{en  ■  -  1/2}  -  P(cn  •  1/2}  ■  1/2.  Solving  the  recursion  (6.4) 
yields 


Xn+1  *  (2J  en+l-k  +  V  *0  * 

Hence,  for  -1  <  x  <  1, 

(6.5)  Ex  exp(itXQ+^) 

-  exp(2_(n+1)  itx)  •  n  E  exp(2"k  it  e  ..  .  ) 

k«0 

-  exp(2_(rrt"1)  itx)  •  n  E  exp(2*k  it  e.  ) 

k-0  K 
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where  Che  exchangeability  of  Che  sequence  (e^:  k  >  1)  wcs  used  in 
Che  last  equality.  Taking  limits  in  (6.5)  gives 

0  CD 

lim  E  exp(itX  )  -  n  E  exp(2  ^  ite.)  *  n  cos(2  ^  £)  -  sin  t/t 
a  +  »  x  n  k-0  K  k-0 

(see  [4],  p.  165,  for  Che  infinite  product  closed  form).  The 
continuity  theorem  for  characteristic  functions  can  now  be  applied  to 
show  that 


(6.6)  X  — >  D  P  -weakly 

n  x 

for  all  x  c  (-1,IJ,  where  0  is  a  uniform  random  variable  on 
(-1,1].  By  Lemma  6.2,  it  follows  that  x(dx)  -  dx/2  is  Invariant  for 
the  chain.  Furthermore,  since  the  weak  convergence  in  (6.6)  occurs 
for  all  x,  x(dx)  is  the  unique  invariant  probability  for  {Xq}. 

Suppose  now  that  {Xq},  when  appropriately  restricted,  defines 
a  well-posed  simulation  problem.  Then,  there  must  exist  a  point 
x  c  E  such  that 

,  n  1 

(6.7)  7^r?T  I  <p  f)(x)  -  I  £(y)  dy/2  ,  for  all  f  e  bE  . 

^  k-0  -1 

Now,  observe  that  when  the  chain  starts  at  x,  the  process  {Xq} 
thereafter  takes  values  in  the  countable  set 
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A  -  { 2~*x  +  J2-k;  A,  J,  k  >  0}  . 

Thus,  putting  f  -  1^,  we  see  that 

T5TTT  j0  yXfe  «  A)  -  1  >  0  -  /  iy/2  , 

contradicting  (6.7).  Hence,  the  process  has  no  well-behaved 
restriction.  Note  that  in  light  of  Theorem  5.2,  this  also  implies 
that  {Xn}  possesses  no  initial  distribution  p.  under  which 
{Xn}  can  be  embedded  in  a  weakly  regenerative  environment. 

The  difficulty  here  lies  in  the  class  of  functions  for  which 
convergence  holds.  In  the  well-posed  case,  it  is  assumed  that 

(6.S)  y  J  (Pkf)(x)  *  *f 

for  all  f  c  bE,  whereas  the  autoregressive  example  above  is  typical 
of  the  case  where  (6.8)  holds  only  for  f  c  bC.  One  might  call  such  a 
case  weakly  well-posed  since  (6.8)  then  is  equivalent  to  the  weak 
convergence  statement 


(nil)  pk(x»#)  —>*(•)  • 


The  conclusion  is  therefore  that  assumption  of  only  weak  convergence 
of  the  averaged  transition  probabilities  can  lead  to  processes  that 


ja. 


:  V 
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exhibit  quite  different  behavior  from  that  which  one  might  expect  from 
"well-behaved”  chains. 

From  a  simulation  standpoint,  a  weakly  well-posed  problem  has 
several  deficiencies.  First  of  all,  convergence  in  (6.8)  is 
guaranteed  only  for  continuous  functions  f,  and  therefore  problems 
may  arise  with  Indicators  (indicators  arise  frequently,  in  practice). 
Secondly,  for  discontinuous  f,  it  may  be  that 

rQ(f)  +  r(f)  a.s. 

and  yet  r(f)  *  itf.  For  such  an  f,  a  simulator  would  have  a  tendency 
to  incorrectly  interpret  rn(f)  as  uf,  while  the  simulation  would 
give  no  Indication  of  bad  behavior. 

7.  Smoothness  of  the  Kernel  and  its  Relation  to  Well-posedness 

In  Section  6,  we  showed  that  Markov  chains  with  unique  invariant 
probability  can  not  necessarily  be  restricted  in  such  a  way  so  as  to 
produce  a  chain  for  which  the  simulation  problem  is  well-posed.  It  is 
therefore  of  interest  to  produce  necessary  and  sufficient  conditions 
which  guarantee  existence  of  such  a  restriction.  Our  next  theorem 
addresses  this  question. 

(7.1)  THEOREM.  Suppose  that  {Zq}  has  a  unique  Invariant  prob¬ 
ability  x*  Then,  the  following  are  equivalent: 

1)  the  chain  {Zq}  possesses  a  stochastically  closed  set  A, 
with  x(A)  >1,  such  that  the  steady-state  simulation  problem 
is  well-posed  for  the  process  restricted  to  A. 
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11)  there  exist*  a  non-trivial  a— finite  Measure  v  such  that  the 
sat 

(7.2)  B  -  {x:  v(»)  <  R(x,»)} 

has  positive  x -measure  «  denotes  "absolutely  continuous 
with  respect  to"),  where  R(x,»)  ■  ^  2"*  P^Cx, •). 

Before  proving  the  result,  we  remark  that  B  is  an  E-measurable 
subset  of  E  (see  REVUZ  (1975),  p.  78). 

Proof.  Under  (1),  the  process  restricted  to  A  is  Harris 

r 

A  A 

x-recurrent  (Proposition  3.8),  where  it  is  an  invariant  probability 
for  the  process  restricted  to  A.  In  fact,  because  A  Is  stochas- 

A 

tically  closed,  it  must  also  be  Invariant  for  the  unrestricted  chain, 
and  hence  it  »  x.  In  any  case,  for  all  x  e  A,  it(D)  >  0  Implies  that 

(7.3)  l  IjjOy  -  -  Px  a. s. 

k-1 

Taking  expectations  in  (7.3)  proves  that  R(x,D)  >  0  and  thus  (11) 
holds  with  v  *  it  and  B  -  A. 

For  the  converse,  observe  that  the  shift  9  is  a  measure¬ 
preserving  ergodic  transformation  on  (QfFtP*)  (the  ergodldty 
follows  from  the  uniqueness  of  x;  see  [1],  p.  141),  and  hence 
Blrkhoff's  ergodic  theorem  applies.  Thus,  it(D)  >  0  implies 

OB 

(7.4)  l  I  (X.  )  -  »  P  a.s. 

k-1  u  *  x 
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for  *  a.e.  x.  Taking  expectationa  in  (7.4)  shows  that  n(D)  >  0 
forces  R(*,D)  >  0  *  a.e.,  and  so  (Xq)  is  x-essentlally 
irreducible  (see  122],  p.  78,  for  definitions  and  results).  Now, 
observe  that 


v( • )  <  /  R(x, •)  it(dx) 

B 

by  (7.2).  On  the  other  hand, 

/  R(x«)  x(dx)  <  /  R(x, •)  it(dx)  *  it(«) 

B  E 

and  hence  v  <£  it.  Therefore,  on  the  set  B,  the  measures  *(•)  and 
R(x,»)  are  not  mutually  singular  (they  have  a  v-component  in 
common),  and  so,  by  Theorem  2.14  on  p.  78  of  122],  the  chain  {XiJ 
possesses  a  stochastically  closed  subset  c  E  such  that  x(A^)  *  1, 
and  the  restriction  to  is  x-irreducible  (this  means  that  x(B)  >  0 
implies  that  R(x,B)  >  0  for  all  x  c  A}).  Let  G  be  the  poten¬ 
tial  kernel  given  by 


G(x,» )  -  l  Pk(x,») 
k-0 

for  x  e  Aj.  He  will  now  show  that  G  is  not  a  proper  kernel.  The 

kernel  G  is  proper  if  there  exists  a  sequence  of  sets  B- 

r 

increasing  to  Aj  such  that  G(*  ,Dn)  is  bounded.  But  n(Dn) 
must  be  positive  for  some  n,  and  thus,  by  (7.4),  it  must  be  that 
G(» ,Dn)  ■  •  x  a.e.,  contradicting  the  properness. 
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We  now  apply  Theorem  2.6  on  page  73  of  [22] ,  which  shows  Chat  Che 
restriction  to  Aj  Itself  possesses  a  stochastically  closed  subset 
A  with  x(A)  ■  it(A})  “  1  such  that  the  restriction  to  A  is 
Harris  n-recurrent.  Since  x  Is  clearly  invariant  for  the  restric¬ 
tion  to  A,  one  can  apply  Theorem  3.8  to  complete  Che  proof.  I 

One  may  interpret  condition  (7.2)  as  a  kind  of  smoothness  condi¬ 
tion  on  the  kernel;  the  kernel  P  must  not  be  "too  singular."  We  see 
that  the  autoregressive  process  example  of  Section  6  fails  to  possess 
a  well-behaved  restriction  in  the  most  general  possible  way,  namely  by 
violating  the  smoothness  condition  on  the  kernel. 
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